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ABSTRACT 


This qualitative research reports findings on whether students’ reflective writings during the course 
of one semester produced qualitative differences in several courses offered online and on-ground at two 
different universities. Eighty-six students from two universities responded to Brookfield’s (1995, 1998) 
critical incident questionnaire. Because of the descriptive nature of our research design, our study utilized 
a qualitative descriptive methodology for the purpose of data analysis. The basic findings of our research 
indicate that reflective practice enables students to think reflectively upon course expectations. Engaging 
students in reflective practices during the course of a semester helps faculty gauge students’ engagement 
or disengagement with course materials. 
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The purpose of this qualitative research was to 
investigate the role of critical reflection for online 
and traditional courses taught at two universities. 
We used cognitive and socio-cultural theories as 
the theoretical lenses for our study. There were two 
primary research questions guiding our research: 
(1) what type of reflective writings would emerge 
from the participants’ weekly reflective logs? (2) 
Would there be significant qualitative differences 
in participants’ reflections from week to week? 

Eighty-three graduate and undergraduate 
students enrolled at two different regional 
universities in the Southeastern U. S. participated 
in the study over one semester. Participants 
responded to the critical incident questionnaire 
(CIQ) in online and face-to-face classes. The data 
were analyzed using the technique of constant 
comparative methodology. The next section of the 
manuscript discusses the background of the study, 
methodology, and findings and conclusions. 

BACKGROUND 

Many universities in the United States use end- 


of-the-semester course evaluations from students 
as a way to collect data on matters concerning 
learning, management, and overall evaluation of 
a specific course offered. These evaluations may 
not be very effective in showing what a student has 
learned during the course of one semester. There¬ 
fore, instructors may find difficulty in capturing 
possible disconnects between course materials and 
student learning using only course evaluations. 
Faculty members need more immediate access to 
student feedback, particularly in preservice teach¬ 
er education courses. Parker (1998) indicated that 
grading and other course structures often do not 
allow changes within the semester. The challenge, 
therefore, in both online and traditional courses, 
is to create and organize course material to pro¬ 
mote more student participation and reflection in 
a course. 

One way to supplement course evaluations is to 
use critical incident questionnaires (CIQ). The CIQ 
can help highlight participation in class activities 
that may help or impede learning. Brookfield 
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(1998) explained that the CIQ captures moments or 
actions in class and reveals student judgments of 
those activities as they learn. The critical incident 
technique has a long history beginning with the 
seminal work of Flanagan (1954) in the United 
States Air Force. Flanagan’s technique included 
the following: (1) setting the aim of an activity, (2) 
creating a tool to collect incident, (3) gathering data 
using either interviews or written observations, 
(4) analyzing data in an objective way, and (5) 
interpreting results and disseminating those that 
make significant impact on the activity. 

While the CIQ has been used in education, 
researchers have also used the technique to 
collect data in fields ranging from health services, 
engineering, sport, and business. For example, 
Hoffman and Chung (1999) utilized the CIQ to 
evaluate hotel services, while Urquhart et al. (2003) 
employed it in their assessment of the application 
of information behavior in Health Service Library 
Projects in Britain. Radford (2006) evaluated 
citywide library projects for Brooklyn Public 
Library and Queens Borough Public Library in 
New York with the technique. Critical incidents 
have also been utilized by researchers in the health 
profession (Bradley, 1992; Schluter, Seaton, & 
Chaboyer, 2008). More recently, Hardin, Ruihley 
and Verlado (2013) examined game day experiences 
in college football using critical incidents, while 
Milner, Ostemeier and Franke (2013) examined 
cross-cultural aspects of coaching using critical 
incidents. 

An important argument for the use of critical 
incidents comes from the current business world. 
Berger, Stratton, Thomas and Cook (2012) argued 
that critical incidents meet the “demand for 
this new genre of shorter, more focused cases” 
in business education (p.2). Similarly, Hanson 
and Brophy (2012) argued for the use of critical 
incidents as “an established method for cognitive 
task analysis” (p. 1). 

Because analyzing critical incidents is a 
reflective technique, it is particularly useful in 
teacher education. Brookfield (1998) indicated that 
critical reflection is about a “process,” a deliberate 
act of knowing about one’s trade and “discovering” 
about the assumptions relating to the teaching 
profession (p. 197). Brookfield (1998) suggested 
that practitioners should evaluate their practices 
through “multiple lenses” (p.197) because people 


learn about their professional practices through 
multiple experiences and perspectives. Therefore, a 
reflective practitioner through the process realizes 
many factors that go into creating the craft of 
teaching. A reflective practitioner reflects through 
their own perspectives, the perspectives of their 
students, and the body of literature or theories that 
inform our practice as teachers. Brookfield (1998) 
calls these “complementary lenses” (p. 197). 

Self-reflection of the practitioner and the 
students under training can be possible and effective 
in understanding skills to be learned (Goodell, 
2006). It shows how teachers at times can negotiate 
difficult situations with students and value of their 
personal thoughts in learning professional skills 
within the classroom (Lin, 2011; Mento & Desai, 
2012). Khandelwel (2009) used critical incidents 
in understanding teachers’ classroom behavior in 
higher education, while Andreou, McIntosh, Ross, 
and Kahn (2015) investigated the experience of 
educators in efforts to sustain school-wide positive 
behavior after training. 

Research in the field of reflective practices 
documents students’ positive responses when 
engaged in an open learning environment where 
they can discuss class activities. For example, 
Cacciamani, Cesareni, Martini, Ferrini, and 
Fujita (2012) reported that students’ metacognitive 
reflection space aided participation and building of 
knowledge. Cacciamani et al. studied 64 students 
using blended learning environments. Their 
findings indicated reflection enabled students 
to develop skills that enhanced their classroom 
experiences and the creation of new knowledge. 
Other researchers agree that it is possible to 
foster reflection in students and gauge progress or 
growth. For example, instructional devices, such as 
writing online or having students create reflective 
journals or self-reports, assist students with their 
development of critical thinking and respective 
professions skills (Boyer, Maher, & Kirkman, 
2006; Chimera, 2006; Hanson & Brophy, 2012; 
Kitchenham, 2006; Trepal & Hammer, 2014). 

PURPOSE 

The purpose of this qualitative descriptive 
research was to investigate the utilization 
of reflective practice using critical incident 
questionnaire in order to collect qualitative data 
that reflects learners’ perceptions of course material 
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throughout a semester. Edvardsson and Roos (2001) 
indicated that data collection using critical incidents 
is possible through a variety of approaches ranging 
from interviews to direct observations. Brookfield 
(1998) argued that critical incidents can be used to 
collect data that reflects a professional’s work. The 
data could be through the lens of others (colleagues 
and learners) or through one’s own stories. The 
process and the data collection can help unearth 
assumptions one makes about own practices that 
may be distorted or may not be useful according 
to self-knowledge. The assumptions of the present 
research were that critical incidents would enable 
instructors to learn new information evidences 
about their actions as they teach and that the CIQ 
would reveal how students perceive or interpret 
course materials. 

THEORETICAL FRAMEWORK 

This research emphasizes the process of 
reflection, and therefore, we utilized several 
complementary theoretical frameworks. Learning 
to create new knowledge involves interpreting 
experiences through multiple modalities, but this 
process in constructivism is not always linear. 
In fact, Piaget (1966, 1970a, 1970b) argued that 
individuals acquire new knowledge by the process 
of fitting new information with old information, 
a process coined disequilibrium, or a state of 
confusion experienced by the learner. 

Actions can be understood as modus operandi 
from which individuals engage and are engaged in 
the process of creating new knowledge. Learning 
in constructivist theories indicates that individuals 
seek knowledge actively. Thus, knowing does not 
exclusively involve the mind or cognition, but the 
mind is used as a tool to mediate prior knowledge 
with novel information in order to create new 
knowledge through adaptation, assimilation, and 
accommodation (Piaget 1966, 1970a, 1970b). Also, 
knowledge is embedded in culture and language, 
so that individuals learn by engaging with and 
becoming engaged by others (Driver, Asoko, 
Leach, Mortimer, & Scott, 1994; Losnot, 1996; 
Shuell, 1986; von Glasersfeld, 1996). 

The other theoretical lens is adult learning 
theory because adults learn in distinct ways. 
How adults learn is different from learning in 
children and animals (Knowles, 1973, 1980, 1984; 
Knowles, Holton, & Swanson, 2012). According 


to Merriam, Caffarella, and Baumgartner (2007), 
adult learning models indicate diverse ways that 
adults learn; adult learning models suggest adults 
learn via self-directed attempts, transformative 
approaches, and experiences. 

Using these theoretical frameworks, students 
in both online and traditional settings create their 
own knowledge by actively pursuing meaning 
constructively. Additionally, we theorize that 
individual reflection by modus operandi should be 
considered a constructive act in light of the fact 
that it is knowledge being created by the individual 
reflection in action. We feel that constructivism as 
a theoretical lens grounds our study well because 
the nature of learning is an individual and social 
act mediated by experiences. 

METHODOLOGY 

We utilized a qualitative descriptive- 
interpretive design for our study since our goal 
was to understand students’ perspectives of their 
class activities. Qualitative descriptive-interpretive 
design employs less detailed interviews that are 
not typical of other qualitative research designs 
(Elliott & Timulak, 2005; Sandelowski, 2000). 
The data gathering in descriptive-interpretive 
design is flexible. Brookfield’s (1995) CIQ allowed 
such data gathering in the form of students’ quick 
responses after classes or at the end of every four 
weeks. The CIQ consists of five questions aimed at 
what engaged, distanced, affirmed, surprised and 
confused the participants during class activities. 
Second, the rationale for using descriptive- 
interpretive design was that it did not need 
additional data analysis techniques to categorize 
or conceptualize data. While the CIQ already 
categorizes comments according to listed themes, 
we also conducted constant comparative analysis 
to generate additional categories that captured a 
“recurrent pattern” of the types of activities that 
affected the participants (Merriam, 1998) which 
emerged from across “the preponderance” (Taylor 
& Bogdan, 1984), and of the responses (Creswell, 
2015; Glaser & Strauss, 1999; LeCompte, Preissle, 
& Tesch, 1993). 

The CIQ lists the five questions with sufficient 
space under each question for the students to write 
the critical moments or actions in the class as they 
perceived and judged them. Eighty-six participants 
responded to the five questions freely after class 
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sessions every four weeks for a total of 15 weeks. 
Participants were instructed not to write their names 
or anything that identified them on the sheets. 
Anonymity, as Brookfield (1998) argues, ensures 
reliability of data because consistent sincerity of 
the instructor distinguishes the professional as an 
entity from his actions and his performance. 

We read and analyzed the data immediately for 
both online and on-ground classes. For the purpose 
of systematic analysis of the responses from the 
participants, we read and analyzed data from 
each class and highlighted phrases that indicated 
situations aligned with each question. We then used 
data logs to enter phrases and words that specified a 
situation or incident in the class for further analysis 
of data and themes according to the questionnaire. 

RESULTS 

The CIQ categorizes responses according to the 
following themes: engaged, distanced, affirmed, 
confused, and surprised. These recurrent themes 


reflected the participants’ thinking about the 
course activities. The findings for online and on¬ 
ground courses were different, but the responses 
from week to week did not vary much for any of the 
classes. In addition, responses were occasionally 
inconsistent. For example, instructor’s reviews of 
previous sessions and discussions worked for some 
and distanced others from learning. Participants 
also appeared to perceive their class activities 
differently. Typical comments from the participants 
are as follows: 

• “Feel engaged when responding to the 
journals written by classmates...” 

• “Engaged when reading reflections on 
experiences of other learners...” 

• “Most engaged when reading...” 
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Table 1 

Themes and Corresponding Responses from Students 


Themes 

Response On ground 

Response Online 

Engaged 

During discussions 

Thoughts and ideas from my peers 

When presenting our research 

In groups reasoning outforanswers 

When learning what I can use as a teacher 

Discussion after classvideos 

Team projects 

When going over new information 

Whenrespondingtojournals 

Distanced 

Going over questions from the text book 

Discussing scenarios from the text. 

When ideas are not engaging and making me interact 

When discussing new concepts or content. 

Lectures 

Reading certain chapters in the textbook 

When it was time to read articles in class 

When instructor does not confirm the information and I get 
confused 

Doing assignments because I have 3 classes to deal with 

Affirmed 

Going over course materials and getting different 
perspectives 

Discussing articles we bring to class as a whole class 

Exercises where we come up with ideas 

Useful conceptsI can use 

When we worked as group 

Giving examples or explaining something I don’t understand 
Reviewin the beginning of classand hands on activities 

When the instructor writes on the board what we are talking 

about 

When the instructor responses to posts 

When multiple explanations were used from groups and 
instructor 

Confused 

When the discussion strays, it throws me off. 

Astudentasks stuff she should know 

When someone talks over someone else, and I can’t hear 

When the class wanted to negotiate with instructor 

Discussion questions 

Some assignments confuse me 

Surprised 

How integrated subject matter was 

There are many ways of looking at things 

The community that exists between teacher & students and 
among students 

How much easier it is to comprehend with small class size 

How practical this class is Conversation style of the class 
instead of hearing lecture Surprised how well everyone 
worked together. 

Howfun the class discussions are 

Not knowing what I learned before 


Next, we examined the types of situations 
participants mentioned in their responses. The 
categories that emerged were assignments, 
interactions, and intrinsic or personal factors. 
Participants expressed being distanced or engaged 
by their assignments, being distanced or engaged by 
class interactions, and having intrinsic factors unique 
to the individual that affected their engagement. 
Assignment, interactions and personal factors 


influenced additionally the CIT themes of surprise 
and confusion. However, we noted differences in 
student responses to activities in online and on¬ 
ground classes. While 88% of the participants 
reported being engaged by discussions in on-ground 
classes, it was the least engaging method reported 
by online classes in comparison. Online participants 
were also more likely to report distanced by personal 
factors, such as health problems or personal problems. 


GRAND CANYON UNIVERSITY 












75 


Table 2 

Examples of data with categories that showed what distanced students 


Assignments 

Interaction 

Intrinsic 

Individual factors 

Knowing when assignment was due 

Going over questions from the text 
book 

I do take responsibility for my own 
learning 

When ideasare not engagingand 
making me interact 

Reading materials 

Group activities 

If had a question about something, I 
would ask 

Things that haven’t come easy for me 

Being responsible for due dates 

When ideasare not engagingand 
making me interact 

I have energy and clear head 

Making sure I stay focused 

A bunch of assignments due at once. 

When students nearby laugh and 
talk loudly 

I need this class to graduate 

I don’t procrastinate 

Not letting students work in class 
because most of the assignments 
are done outside the class. 

When working with classmates. 

Continue to learn howto do more 
things in thecomputer 

I need detailed content page in the 
site. 




Personal issuegoingon. 




Giving my full attention 


DISCUSSION 

The rationale of this qualitative descriptive 
research was to examine whether critical incidents 
would emerge from students’ weekly reports and 
how students interpreted class activities. The CIQ 
indicated online participants were engaged when 
they read course materials, posted online, and when 
reading other participants’ journals and postings. 
However, the on-ground participants’ comments 
showed they were engaged during class discussions 
and in collaborative groups. Recurring comments 
were engaged when in “groups” or “when talking 
to people.” Participants in on-ground classes also 
reported responding to class activities well when 
the instructor reviewed materials and wrote on 
white-board or introduced new material. 

This study supports previous findings that 
reflection through classroom experiences such as 
writing online or asking students to create reflective 
journals helps keep students engaged in learning 
(Boyer, Maher, & Kirkman, 2006; Cacciamani 
et al., 2012; Chimera, 2006; Kitchenham, 
2006). However, in this study, online and on¬ 
ground participants differed in how they viewed 
collaboration in class. While the on-ground class 
viewed it favorably, collaborative groups distanced 
online participants because they reported that some 
of the group members abused the online format 
by not contributing. When the course format and 
materials did not make sense, particularly in the 


beginning of the course, all participants reported 
feeling distanced. This underscores the importance 
of having an organized, easy-to-follow format or 
outline for the online course. However, on-ground 
participants were distanced by abstract discussions 
of theories and principles discussed during class 
time. On-ground participants reported being 
distanced by personal matters like lack of sleep 
the previous night or by other students distracting 
them in class. 

Online participants stated that instructor 
comments affirmed their learning and resolved 
issues. They said reading the text was both 
affirming and surprising for them, depending on 
their interest or relevance of the topics discussed 
in the textbook. However, on-ground participants 
said they were more engaged and interested in 
practical assignments and reviews. For on-ground 
participants, reading and discussing the textbook 
seemed to distance them from learning. Reading 
seems to be a critical source of divergent views for 
participants in both learning environments. While 
reading the textbook engaged online participants, it 
distanced on-ground participants. Extra materials 
brought into the class distanced online participants, 
while it was engaging to on-ground participants. 

Therefore, to address the research questions that 
guided the study, we found, through the CIQ, that 
participants shared their judgments about course 
activities. Most of the participants’ responses were 
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reflections on their individual learning. However, as 
mentioned in the results, the responses did not vary 
much from week to week. This is an indication that 
participants’ impressions of course activities were 
consistent, and they responded to course activities 
and materials in a regular manner. Of course, the 
most striking finding was the difference between 
how helpful participants in online and on-ground 
classes found the textbook and group discussions. 

CONCLUSION AND IMPLICATIONS 

From this study, we concluded that the student 
engagement in the learning situation for on-ground 
participants depended on hands-on activities and 
mostly happened when collaborating with peers. 
For example, on the question of what engaged them, 
they wrote, “Talking to other students, collaborative 
activities, visuals and computers,” or “laidback 
environment and discussions.” On the question of 
when they felt distanced from class activities, they 
said when they did not understand information 
delivered. They wrote, “When discussing new 
concepts or content,” “lectures,” or “reading certain 
chapters in the textbook.” On-ground participants 
also reported being distanced by information 
they thought was not useful to them. Participants’ 
interpretation of class time and class activities did 
not necessarily match instructors’. They indicated 
that they needed to understand why they were 
learning certain concepts, and they were engaged by 
activities beyond what they could learn through the 
textbook. This indicates a different need than what 
online participants expressed. Online participants 
did not express being engaged by supplementary 
material, but they reported being engaged by the 
activities they did themselves, such as reading and 
responding to the textbook. The implications of 
this finding are important for instructors of online 
classes. While supplementary materials may be 
provided with the goal of enhancing the online 
experience, students may not view them that way. 
They were most engaged with materials that were 
required for completion of coursework, not those 
they thought were ancillary. 

One of the limitations of this study is that it 
is restricted to a small sample of students. The 
findings are not necessarily generalizable outside 
the scope of this qualitative research. The next step 
would be to replicate the study using a quantitative 
or mixed methods research design. This qualitative 


study’s aim was on gaining knowledge from 
participants’ experiences as opposed to predicting 
behavior. Nevertheless, we think that our results 
are promising because they tap into issues 
relating to student learning and engagement in 
course materials in online and traditional courses. 
Additional research is needed on this topic to 
explore possible causes of the difference in student 
engagement in virtual and campus-based classes. 
We think that both qualitative and quantitative 
empirical studies are necessary in order to gain 
additional knowledge on students’ engagement in 
learning activities. 

We recommend further inquiry applying the 
technique. Different methodologies can be used as 
flexible designs to explore different facets of and 
issues in students’ learning. Multiple case studies 
could be useful because of their ability to compare 
findings across cases (Baxter & Jack, 2008; Yin, 
2003). They also have the potential, in their results, 
to develop models to implement in teaching and 
learning. 
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